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Abstract 

The    literature    exploring    computer-based    information    system    (CBIS) 
evaluation  has  explicated  a  number  of  methods  for  investigating  the  impact  of 
systems  on  organizations.  However,  while  CBIS  evaluation  methods  have  been 
scrutinized  and   debated,  the   assumptions  embedded   in  differing   evaluative 
approaches  have  remained  largely  tacit.  Highlighting  the  ontological  (about  the 
nature  of  reality)  and  epistemological  (about  the  nature  of  valid  knowledge) 
assumptions  of  alternative  evaluative  methods  suggests  two  paradigms  for  CBIS 
evaluation  -  positivist  and  interpretive.  Positivist  CBIS  evaluation  is  characterized 
by  detachment  of  evaluator  from  subject,  a  priori  categorization,  and  a  quest  for 
generalizable    knowledge.    Interpretive    CBIS    evaluation    is    characterized    by 
experiential    involvement    of  researcher    in   subject,    an    absence    of   a   priori 
classification,  and  a  quest  for  accurate  descriptions  of  particular  circumstances. 
Recognizing  the  ontological  and  epistemological  differences  between  positivist 
and  interpretive  evaluation  suggests  four  propositions  useful  in  improving  CBIS 
evaluation  research.  These  propositions  include  acknowledging  values,  beliefs, 
and    assumptions    implicit    in    CBIS    evaluation,    recognizing    and    discussing 
evaluation    trade    offs,    building    'complicated'    descriptions    of    CBISs,    and 
recognizing    the    interdependence    of    positivist    and    interpretive    evaluation 
approaches. 


An  organization  implements  a  computer-based  information  system  (CBIS). 
Later,  someone  asks  if  the  CBIS  is  a  success  or  failure.  Does  it  contribute  to 
organizational  goals?  Should  it  be  maintained,  expanded,  replaced,  or 
abandoned? 

Recent  research  suggests  that  a  major  obstacle  to  investment  in  new 
CBISs  is  the  inability  to  measure  and  evaluate  productivity  gains  resulting  from 
such  investments  (Blacker  and  Brown,  1988;  Strassman,  1985).  Controversy  over 
measuring   productivity    contributions   from    new   technology    has    resulted    in 
increasing  skepticism  regarding  the  benefits  of  CBIS  in  general  (Business  Week, 
1988),  and  microcomputers  (Bowen,  1986)  and  office  automation  (Strassman, 
1985;  Uttal,  1982)  in  particular.  This  controversy  over  CBIS  productivity  mandates 
a  reexamination  of  CBIS  evaluative  practice.  Specifically,  how  can  we  determine 
why  and  when  CBISs  contribute  to  managerial  productivity? 

IS  researchers  have  long  recognized  the  importance  and  complexity  of  this 
and  related  questions.  As  a  result,  a  number  of  methods  for  evaluating  CBIS  have 
evolved.  However,  it  is  not  generally  recognized  that  methods  of  evaluating  CBIS 
reflect  basic  values,   beliefs,   and   assumptions   about  what   constitutes  valid 
knowledge.  While  the  methods  for  CBIS  evaluation  have  been  scrutinized  and 
debated  (e.g.,  Hamilton  and  Chervany,  1981a;  Sprague  and  Carlson,  1982),  the 
assumptions  embedded  in  differing  evaluative  approaches  have  remained  largely 
tacit.  However,  the  hidden  values  and  assumptions  implicit  in  CBIS  evaluation 
methods  can  impede  understanding  of  the  benefits  and  limitations  of  alternative 
evaluation  practices  (Hirschheim  and  Smithson,  1988).  IS  researchers  are  less 
likely  to  trip  over  assumptions  implicit  in  evaluative  methods  if  these  assumptions 
are  highlighted,  discussed,  and  debated.  The  purpose  of  this  paper  is  to  highlight 
differing  epistemologies  used  in  evaluation,  and  to  relate  these  epistemological 
differences  to  CBIS  evaluation  practice. 
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One   important   approach   to   improving   CBIS   evaluation    has   been   to 
explicate  alternative  methods  of  CBIS  evaluation  (Hamilton  and  Chervany,  1981a, 
198b;  Sprague  and  Carlson,  1982;  Srinivasan,  1985),  and  to  delineate  differing 
paradigmatic    approaches    to    evaluation    (Hirschheim    and    Smithson,    1988). 
Another  important  line  of  research  has  examined  conditions  in  which  CBISs  are 
incorporated  into  organizations  (Kling  and  Iacono,  1984;  Markus  and  Robey, 
1988;  Zmud  and  Apple,  1989).  These  efforts  provide  a  foundation  for  under- 
standing the   fundamental  relationships  between  information   technology   and 
organizational  change,  and  for  linking  theories  of  evaluation  with  more  general 
scientific  principles.  This  paper  complements  these  efforts  by  explicating  the 
paradigmatic  assumptions  of  alternative  theoretical  perspectives  of  evaluation, 
and  by  linking  these  assumptions  with  CBIS  evaluation  methods. 
I.  CBIS  EVALUATION 

CBIS  evaluation  is  herein  defined  as  the  process  of  formally  determining 
how  a  CBIS  impacts  and  is  impacted  by  an  organization.  Several  assumptions  are 
implicit  in  this  definition.  First,  that  a  CBIS  has  been  implemented,  meaning  that 
evaluation  is  a  post-implementation  activity.  Evaluation  is  therefore  identified  as 
distinct  from  feasibility  analysis  (Caddell,  1985)  and  a  priori  justification  of  CBIS 
(Bozcany,  1983;  Gremillion  and  Pyburn,  1985).  Second,  formal  processes  are 
used  to  evaluate  the  CBIS.  While  these  formal  processes  may  be  qualitative,  they 
are  systematic  and  not  purely  impressionistic,  off-hand  reflections.  Third,  it  is 
assumed  that  organizations  both  create  and  are  created  by  CBIS.  Evidence 
suggests  that   the   process   of  implementing  a   CBIS   creates   complex,   often 
unanticipated  chains  of  events  in  organizations  (Barley;  1986;  Markus  and  Robey, 
1988).  These  chains  of  events  ultimately  mean  that  organizations  create  CBISs 
and  CBISs  create  organizations.  Finally,  it  is  assumed  that  significant  linkages 
exist  between  evaluation  methodologies  and  theoretical   models  (Markus  and 
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Robey,  1988;  Weick  1984).  One's  choice  of  a  method  is  colored  by  implicit 
theoretical  values  and  biases.  As  a  result,  the  choice  of  a  CBIS  evaluative 
methodology  is  also  frequently  an  implicit  choice  of  a  theoretical  perspective 
(since  "believing  is  seeing,"  Weick,  1979,  p.  135). 

Why  do  organizations  evaluate  CBIS?  Research  suggests  that  evaluations 
contain  dual  relevance  to  organizations1.  Evaluation  is  useful  in  producing  valid 
knowledge  by  probing  and  sensing  internal  organizational  realities  (Patton,  1987). 
However,  evaluation  is  also  an  important  component  of  the  authority  structure  of 
an  organization  (Scott,  Dornbusch,  Busching,  and  Laing,  1967).  As  a  component 
of  the   organizational   authority   structure,   evaluations   play   a   crucial    role    in 
administering  rewards  and  punishments  to  organizational  actors  (Pfeffer,  1978). 
The  dualistic  nature  of  evaluations  causes  evaluative  processes  to  both  impact 
and  reflect  the  more  general  interplay  between  rational  and  political  processes  in 
organizations.  As  a  result,  alternative  approaches  to  evaluating  CBISs  reflect  not 
only  alternative  modes  and  methods  of  acquiring  valid  knowledge,  but  also  al- 
ternative methods  for  legitimating  political  activity  (Legge,  1984),  for  signalling  and 
symbolizing  rationality  and  competence  (Feldman  and  March,   1981),  and  for 
managing  and  manipulating  organizational  actors  (Hirschheim  and  Smithson, 
1988). 

The  role  of  evaluation  in  producing  valid  knowledge  can  be  conceived  as 
an  application  of  general  scientific  principles  of  inquiry  and  knowledge  (Legge, 
1984).  Kuhn's  concept  of  "paradigm,"  while  the  focus  of  considerable  debate,  is 
useful  in  delineating  differing  modes  of  inquiry  used  in  evaluating  CBIS.  As 


The  dual  nature  of  evaluative  activity  exists  regardless  of  relationships  between  evaluator  and 
organization.  Academicians,  who  are  relatively  independent  of  the  organizations  whose  systems 
they  evaluate,  are  generally  concerned  about  both  contributing  to  the  body  of  research  knowledge 
and  acquiring  status,  resources,  and  security  within  the  research  community  (Whitley,  1984).  IS 
professionals  who  evaluate  CBISs  are  generally  concerned  about  both  learning  the  impact  of 
specific  systems  and  using  the  evaluation  process  to  further  political  ends. 
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conceived  by  Kuhn  (1970)  and  expanded  by  others  (e.g.,  DeMey,  1982;  Toulmin, 

1972),  a  paradigm  is  not  a  synonym  for  theory,  but  rather  identifies  the  mosaic  of 

theoretical  frameworks,  methodology,  and  ideology  that  comprise  a  discipline. 

Legge,  building  upon  the  work  of  Burrell  and  Morgan  (1979),  suggests  that 

choices  of  evaluation  methods  follow  directly  from  paradigmatic  differences: 

A  paradigm,  as  defined  by  Kuhn  (1970)  is  a  set  of  interrelated 
assumptions  about  the  social  world  which  provides  a  philosophical 
and  conceptual  framework  for  the  systematic  study  of  that  world. 
These  assumptions  are  ontological  (about  the  nature  of  reality), 
epistemological   (about   the   grounds   of  knowledge,   about   how 
people  know,  what  they  know  and  what  the  limits  of  that  knowledge 
might  be)  and  methodological  (about  the  methods  to  be  employed 
to  gain  knowledge  about  reality).  The  methodological  assumptions 
essentially  are  derived  from  the  epistemological,  which,  in  turn,  are 
derived  from  the  ontological  assumptions  of  the  researcher.  Hence 
any  evaluation  design  derives  basically  from  a  particular  world  view 
held  by  the  evaluator  (Legge,  1984,  p.  74). 

Viewing  the  role  of  evaluation  in  producing  valid  knowledge  from  the 
context  of  general  principles  of  scientific  inquiry  suggests  two  basic  methods  of 
inquiry  or  "paradigms".  Legge  identifies  the  basic  paradigms  of  evaluation  as 
positivist  and  interpretive.  These  paradigms  contain  fundamentally  differing  sets  of 
values,    beliefs    and    assumptions    that    are    reflected    in    differing    methods, 
approaches,  and  ideologies  of  CBIS  evaluation.  While  related  dichotomies  have 
been  discussed  in  the  CBIS  evaluation  literature2,  previous  research  has  not 
focused  on  explicating  the  assumptions  underlying  differing  evaluative  paradigms, 
or  on  the  implicit  linkages  between  paradigmatic  assumptions  and  evaluation 
methodologies.  Such  is  the  intent  of  this  paper. 


2  i 

Researchers  have  identified  formative  versus  summative  evaluation  (Hamilton  and  Chervany 

1981a,  1981b;  Meals,  1977),  analytical  versus  interpretive  evaluation  (Hirschheim  and  Smithson, 

1988),  discovery  versus  testing  research  (Franz  and  Robey,  1987),  and  positivist  and  process 

research  (Blacker  and  Brown,  1983). 
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II.  THE  POSITIVIST  AND  INTERPRETIVE  EVALUATION  PARADIGMS 

Positivist  and  interpretive  evaluation  are  both  useful  in  CBIS  evaluation. 
However,   they   approach   problems   with   different   values,    assumptions,    and 
methods,  and  provide  radically  different  evaluation  results.  To  more  clearly  expli- 
cate paradigmatic  differences,  they  are  presented  herein  as  polar  opposites.  In 
reality  however,  there  is  a  wide  spectrum  of  approaches  to  scientific  inquiry  and 
CBIS  evaluation,  many  of  which  combine  paradigmatic  elements  (Morgan  and 
Smircich,    1980).   Hence,   dichotomizing   evaluation  paradigms   represents   an 
oversimplification.  However,  by  explicating  the  extremes  of  different  approaches, 
and  magnifying  their  differences,  a  clearer  picture  emerges  of  the  usefulness  of 
alternative  evaluative  paradigms  in  exploring  CBIS  impact. 

Figure  1  (adapted  from  Eve  red  and  Louis,  1981)  summarizes  the  positivist 
and  interpretive  modes  of  inquiry.  The  vertical  axis  describes  the  paradigm 
(positivist    and    interpretive)    while    the    horizontal    axis    defines    paradigmatic 
characteristics.   The   first   set   of  assumptions   to   be   examined   relate   to   the 
evaluator's  role,  the  evaluator's  relationship  to  the  setting,  and  the  validation  basis 
used  to  determine  the  adequacy  of  knowledge.  In  positivist  evaluation,   the 
evaluator's  role  is  that  of  an  onlooker,  or  outsider  (Legge,  1984).  The  researcher 
views  herself  as  detached  and  separate  from  the  object  under  study,  and  valid 
knowledge  obtains  when  the  evaluator  accurately  mirrors  the  objective  reality  of 
the    phenomena    under    study.    Valid    knowledge    is    obtained    by    accurately 
measuring  the  phenomena  under  study,  analyzing  the  resulting  data  for  logical 
patterns,  and  comparing  empirical  results  with  hypothesized  relationships. 


Insert  Figure  1  about  here 


Underlying  positivist  evaluation's  detachment  of  researcher  from  the  object 
under   study   are    crucial    epistemological    assumptions:    that   there    exists   an 


Figure   1 

Characteristics  of  Positivist  and  Interpretive  Evaluation 
(adapted  from  Evered  and  Louis,   1981) 
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objective  reality  consisting  of  facts  that  have  a  determinate  nature  or  essence  that 
is  knowable.  Beginning  with  Popper  (1972a,  1972b)  however,  philosophers  of 
social  science  have  increasingly  questioned  the  validity  of  viewing  theory  as 
separate  from  observed  phenomena  (Feyerabend,  1975;  Kuhn  1970;  Lakatos 
1970).  However,  critics  of  positivism  argue  that,  "observations  are  fallible 
propositions  which  are  theory-dependent  and  therefore  cannot  act  as  the  neutral 
arbitrator  between  competing  theories"  (Chua,  1986,  p.  612).  Habermas  (1978) 
refers  to  the  assumption  of  a  knowable,  external  world  as  the  "objectivist  illusion." 

Banker  and  Kauffman's  (1988)  study  of  the  impact  of  automated  teller 
machines  (ATMs)  on  bank  profitability  illustrates  positivist  evaluation  assumptions. 
The  evaluators  were  logically  separated  from  the  phenomena  under  study  (i.e., 
the  banks),  and  came  to  "know"  the  impact  of  ATMs  on  bank  profitability  by 
collecting  data  from  an  external,  knowable  world.  Data  provided  by  the  banks 
themselves,  from  U.S.  Census  information,  and  from  a  private  consulting  firm 
were  seen  as  objective  and  independent  of  the  theoretical  propositions  advanced. 
Since  data  and  theory  were  derived  independently,   the   data  consisting  an 
empirical  validation  of  the  "truth"  of  the  theoretical  propositions  regarding  the 
impact  of  ATMs  on  bank  profitability. 

Interpretive    evaluation    views     the     evaluator's     role,     the     evaluator's 
relationship  to  the  setting,  and  the  basis  of  valid  knowledge  very  differently.  In 
interpretive   evaluation,   the   evaluator  is   also   a   participant   in   the   evaluation 
process.   The    researcher   comes   to   know   the   "subject"   by   immersion   and 
involvement  in  the  phenomena  under  study.   Knowledge   is  validated  not  by 
forming  arbitrary  distinctions  between  "theory"  and  "data",  but  by  appeal  to  a 
combination  of  logical  consistency,  subjective  interpretation,  and  agreement  with 
actors'  common-sense  interpretation  (Schutz,  1962). 
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Underlying  interpretive  evaluation  is  a  very  different  set  of  epistemological 
assumptions:  knowledge  results  from  human  experience,  which   is  inherently 
continuous    and    nonlogical,    but    which    can    be    symbolically    represented 
(Heidegger,  1962).  Because  there  exists  no  neutral,  objective  world  of  facts  to  act 
as  final  arbitrator,  evaluators  must  take  seriously  the  social,  subjective  realities  of 
actors  that  emerge  in  human  interaction.  The  hazard  of  such  a  perspective  is  that 
no  clear  standards  exist  for  judging  the  adequacy  of  an  explanation  (Bernstein, 
1976;  Habermas,  1978).  As  a  consequence,  the  findings  of  the  evaluation  may  be 
distorted  and  contaminated  by  the  values  and  beliefs  of  the  evaluator.  Bertram 
Russell  (1945)  refers  to  the  rejection  of  an  objective,  knowable  world  as  the 
"fallacy  of  subjectivism." 

Mann  and  colleagues  (Mann  and  Hoffman,   1960;  Mann  and  Williams, 
1960)  study  of  the  introduction  of  a  transaction  processing  system  (TPS)  in  a 
power   company    illustrates    interpretive    evaluation.    The    investigators    began 
interviews  with  employees  of  the  power  company  six  years  prior  to  introduction  of 
the  new  system,  and  collected  data  for  the  five  years  during  which  the  system  was 
designed  and  implemented.  The  evaluators  came  to  "know"  the  impact  of  the  new 
system  through  over  300  unstructured  interviews  with  employees  at  a  variety  of 
levels  in  the  organization.  Although  the  evaluators  were  not  members  of  the 
organization  under   study,   by   intense,   sustained   observation,   awareness    of 
linguistic   cues,    and    careful    attention    to    detail,    the    investigators    came    to 
understand  what  the  new  TPS  "meant"  to  organizational  actors. 

The  Banker  and  Kauffman,  and  Mann  and  colleague  investigations  also 
reveal    differences    in    paradigmatic    assumptions    regarding    the    sources    of 
analytical  categories   used   to   classify   data   collected   during   evaluations.    In 
positivist  evaluation,  investigators  typically  preselect  categories  used  to  classify 
data,  and  then  generate  hypotheses  or  predictions  from  these  categorizations. 
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For  example,  Banker  and  Kauffman's  (1988)  assertion  that  ATMs  can  be  used  as 
competitive  weapons  by  banks  evidences  a  priori  classifications  regarding  how 
the  data  will  be  analyzed.  Only  data  relevant  to  these  prespecified  categorizations 
were  collected.  Such  prespecification  contains  the  hazard  that  important  results 
will  be  omitted  since  they  fall  outside  of  preclassifications.  At  an  extreme,  a  priori 
classification  can  lead  to  evaluations  that  "discover"  only  what  the  evaluators 
expect  to  discover  (Weick,  1984),  by  providing  data  that  are  hopelessly  biased  by 
the  preconceptions  of  investigators. 

In  contrast,  interpretive  evaluations  begins  with  observations  rather  than 
theories.  Salient  features  of  the  phenomena  under  study  are  uncovered  through 
the  experiential  process  of  exploring  the  social  meanings  and  interactions  of 
organizational    actors.    Data    and    categories    emerge    simultaneously    and 
interactively  as  information  is  interpreted  in  light  of  shared  understandings  evident 
in  the  language  and  ideology  of  the  social  system  under  scrutiny  (Morgan  and 
Smircich,   1980).  However,  categorization  in  interpretive  evaluation  has  been 
criticized  for  being  idiosyncratic  and  nongeneralizable   beyond  the  particular 
evaluative  setting  within  which  it  arose.  At  an  extreme,  interpretive  evaluation 
yields  information  that  is  unusable  beyond  individual  contexts  (Legge,  1984). 

Mann's  study  of  computing  illustrates  an  interpretive  approach  to 
categorization.  Theoretical  propositions  about  the  impact  of  the  system  emerged 
from  the  data,  rather  than  being  a  priori  constructs  to  be  validated  by  the  data. 
Conclusions  are  grounded  in  the  specific  circumstances  found  in  the  evaluative 
setting  explored  by  the  evaluators.  For  example,  significant  changes  in 
procedures,  job  structure,  and  management  attitudes  were  found  to  result  from 
the  introduction  of  the  TPS.  These  findings  were  not  hypothesized,  and  are 
presented  as  being  dependent  upon  the  complex,  interrelated  forces  found  in  the 
organizational  environment  surrounding  the  new  system. 
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The  differing  modes  of  inquiry  also  result  in  fundamentally  different  types  of 
knowledge   acquired.    Positivist    evaluation    generates    generalizable,    universal 
knowledge  or  theoria  (Heidegger,  1962).  Positivist  evaluation  seeks  universal  laws 
or  principles  that  can  be  used  to  explain,  predict,  and  control  events.  Such 
principles  are  evident  in  the  Banker  and  Kauffman  study.  The  use  of  a  particular 
ATM  network  is  associated  with  positive  market  share  (p.  145).  Such  a  finding 
provides  general  knowledge  regarding  the  relationship  between  ATM  networks 
and   profitability.   The   fact   that   positivist   evaluation  may   show   that   such   a 
relationship  is  contingent  upon  other  factors  (e.g.,  effective  marketing  practices), 
merely  means  that  additional  environmental  preconditions  are  necessary  for 
predicting,  explaining,   and  controlling   events.   Even  when  contingencies   are 
necessary  to  produce  a  predicted  result,  positivist  evaluation  approaches  still 
yield  general,  universal  knowledge. 

Interpretive  evaluation  generates  knowledge  that  is  relevant  to  particular 
situations.  Interpretive  evaluation  emphasizes  understanding  events,  rather  than 
predicting  and  controlling  them.  Interpretive  evaluation  can  be  characterized  as 
seeking  praxis  knowledge,  or  the  knowledge  of  how  to  act  in  particular  situations 
(Bernstein,  1971).  Such  knowledge  cannot  exist  independent  of  an  understanding 
of  human  organizations.  Mann  and  colleague's  descriptions  of  implementing  a 
TPS  does  not  generate  generalizable  knowledge  about  implementing  TPSs,  but 
instead  provides  a  detailed  understanding  of  a  particular  implementation.  Such 
knowledge  provides  a  highly  accurate  description  as  a  result  of  the  intense, 
sustained   involvement   of  the   researchers.   However,   the   description      is   of 
questionable     generalizability.     In     contrast,     positivist     evaluation     produces 
generalizable  knowledge  that  is  of  questionable  value  in  understanding  particular 
evaluative  results  and  settings. 
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Assumptions  about  the  nature  and  meaning  of  data  significantly  differ 
between    evaluative    paradigms.    Positivist    evaluation    obtains    generalizable 
knowledge  by  stripping  away  what  is  peculiar  to  individual  organizations  to  reveal 
the  kernel  of  presumed  universal  truth.  Data  are  context-free:  they  have  the  same 
meaning  across  organizational  settings.  Data  are  separated  from  context  by  a 
quantitative     evaluation     language,     that     applies     standardized     sampling, 
aggregation,    and    statistical    analysis    to    strip    away    what    is    unique    about 
organizations  and  reveal  what  is  general  to  them.  Banker  and  Kauffman  therefore 
collect  only  data  that  are  common  to  all  banks  studied,  and  intentionally  ignore 
particular  circumstances  within  individual  banks  that  might  explain  ATM  prof- 
itability or  usefulness. 

Interpretive    evaluation    views    data    as    contextually    embedded    in    the 
complex  fabric  of  political,  social,  and  historical  forces  from  which  it  emerges. 
"Facts"  cannot  be  logically  separated  from  contexts,  since  the  "meaning"  of  such 
"facts"    is    dependent    upon    the    shared    understandings    of    organizational 
participants.  When  the  evaluator  understands  contexts  and  the  nature  of  the 
forces  that  have  produced  them,  then  real  knowledge  obtains.  The  evaluation 
language    in    interpretive    evaluation    places    the    evaluator    directly    in    the 
organizational  context,  and  emphasizes  obtaining  a  qualitative  understanding  of 
the   perspectives   of  organizational   participants.   Mann's   study   illustrates   the 
interpretive  view  of  contextually-embedded  data  and  meaning.  Mann  spent  eleven 
years  understanding  and  interpreting  the  complex,  interdependent  forces  that 
shaped  computing  in  the  power  company  under  study.  His  data  are  qualitative 
and  designed  to  reveal  the  perspectives  of  individual  organizational  participants, 
rather  than  to  generate  universal  statements  regarding  the  impact  of  computing 
on  organizations. 


The  Epistemology  of  Evaluation 

11 

III.  PARADIGMATIC  LINKAGES  WITH  EVALUATION  METHODOLOGY 

The  evaluation  paradigms  described  in  section  two  logically  link  with  CBIS 
evaluation  methodology.  By  first  considering  evaluation  methodology  in  general, 
the  linkages  are  made  more  prominent,  and  the  dominant  methodologies  are 
more  evident.  Figure  2  (adapted  from  Douglas  (1976)  and  Weick  (1984))  can  be 
characterized  as  a  continuum  of  evaluation  methodologies  that  range  from  pure 
immersion  in  natural  experience  to  highly  controlled  observations.  Although  spe- 
cific orderings  are  debatable,  the  forms  of  observation  near  the  top  and  middle  of 
the  list  are  less  controlled  and  involve  naturalistic,  emergent  inquiry.  Forms  of 
observation  near  the  bottom  of  the  list  are  more  controlled  and  involve  methods  of 
observation  that  use  a  priori  classification. 


Insert  Figure  2  about  here 

Low  and  high  variety  evaluation 

One  important  characteristic  of  an  evaluative  methodology  is  its  capacity 
for  environmental  sensing  or  variety  (Patton,  1987;  Pondy,  1977).  The  law  of 
requisite  variety  states  that,  in  order  to  accurately  sense  a  system,  the  sensing 
mechanism  must  be  at  least  as  complex  as  the  system  (Conant  and  Ashby, 
1970).  Stated  differently,  a  sensing  mechanism  must  contain  variety  in  order  to 
register  variety  in  a  system  under  observation.  For  example,  a  photographer  who 
must  separately  photograph  ten  objects,  each  of  which  is  a  different  distance  from 
the  camera,  must  have  a  camera  with  at  least  ten  distinct  settings,  if  all  of  the  pho- 
tographs are  to  appear  uniformly  sharp  (Weick,  1979).  If  the  camera  has  fewer 
than  ten  settings,  then  it  possesses  insufficient  variety  for  the  task. 

Methods  in  the  lower  half  of  Figure  3  are  of  low  variety,  while  those  in  the 
upper  half  are  of  high  variety.  Low  variety  evaluation  will  detect,  process,  and 
exhibit  relatively  less  variety  in  CBIS,  since  the  quantitative  language  of  low  variety 


Figure  2 

Continuum  of  Evaluative  Methodologies 
(Adapted  from  Douglas   (1976)  and  Weick  (1984)) 
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methods  contains  a  smaller  pool  of  symbols  than  does  the  qualitative,  "natural" 
language  of  high  variety  evaluation  (Daft  and  Wiginton,   1979;  Patton,   1987; 
Pondy,  1977).  Low  variety  evaluation  converts  both  the  complex  and  the  simple, 
the  ambiguous  and  the  unambiguous  into  the  exact,  unequivocal  languages  of 
statistics  and  mathematics.  In  contrast,  high  variety  methodologies  register  rela- 
tively more  variety,  since  natural,  qualitative  language  contains  a  larger  pool  of 
symbols  for  expressing  ideas.  Natural  language  contains  more  words  than  any 
person  can  or  needs  to  understand  and  assimilate,  resulting  in  countless  possible 
combinations  for  capturing  the  variety  present  in  organizations. 

Low  variety  methodologies  offer  precision  as  an  alternative  to  variety 
(Campbell,    1975).   Low  variety  methods   can  be   used   to   generate   precise, 
unequivocal  statements  about  CBIS  by  using  a  small  set  of  precisely-defined 
symbols.  For  example,  the  definitions  of  a  "t  test"  and  "probability"  are  relatively 
universal.  In  contrast,  agreement  on  the  correct  method  for  determining  the 
meaning  of  CBIS  "success"  is  ambiguous  and  dependent  upon  organizational 
context.  The  quantitative  languages  of  statistics  and  mathematics  provide  low 
variety    evaluation    methodologies    with    precise    symbols    that    are    useful    in 
generating   unambiguous   descriptions   of  systems.    In    contrast,    high   variety 
methods  use  less  precise  natural  language  as  a  medium  of  expression  (Daft  and 
Wiginton,  1979). 

Methods  for  evaluating  CBIS  do  not  exist  independent  of  issues  of 
epistemology  and  ontology.  Further,  the  choice  of  a  method  strongly  influences 
what  is  "found",  since  epistemological  and  ontological  assumptions  are 
embedded  in  evaluation  methods  (Douglas,  1976).  As  a  result,  what  researchers 
"find"  is  strongly  influenced  by  what  they  expect  to  find  (Weick,  1984). 
Consequently,  positivist  CBIS  evaluators  generally  use  low  variety  methods,  since 
they  seek  and  expect  universal,  generalizable  knowledge,  while  interpretive  CBIS 
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evaluators  generally  use  high  variety  methods,  since  they  seek  and  expect 

particular,  contextually-embedded  knowledge. 

Low     variety         methods     provide     positivist     CBIS     evaluators     with 
"preprogrammed"  techniques  designed  to  eliminate  the  biases  of  the  investigator, 
and  to  engender  the  assumed  separation  of  observer  and  observed  (Legge, 
1984).  The  stronger  control  over  observation  present  in  low  variety  methods  are 
related  to  positivist  assumptions  regarding  a  priori  classification  of  observations. 
Low  variety  methods  rely  more  on  theoretical  preconceptions  of  what  is  to  be 
observed,  and  less  on  observation  (Douglas,  1976). 

High  variety  methods  provide  interpretive  CBIS  evaluators  with  methods  for 
"natural"  observation,  intended  to  capture  the  experience  of  organizational  actors. 
Control  over  observations  is  lessened,  since  data  are  expected  to  "tell  their  own 
story."  This  story  will  be  pieced  together  by  the  researcher  through  ex  post 
classification  and  categorization.  High  variety  methods  therefore  rely  more  on 
observation  and  less  upon  theoretical  preconceptions. 

However,  while  methods  are  related  to  epistemological  assumptions,  they 
do  not  map  directly  into  assumptions.  Evaluators  can  and  do  mix  low  variety 
methodology    categories    with    an    interpretive    perspective,    and    high   variety 
methods  with  positivist  perspectives.  For  example,  covert  field  work  conducted  by 
an  evaluator  working  within  a  positivist  paradigm  might  be  used  to  count  the 
number  and  length  of  interactions  between  a  systems  analyst  and  a  system  user. 
The  same  method  used  within  an  interpretive  paradigm  could  be  used  to  explore 
the  subjective  meanings  shared  by  analyst  and  user.  Thus,  while  associations 
between  paradigms  and  methodologies  are  found,  there  does  not  exist  a  one-to- 
one  mapping  of  paradigms  into  methods. 
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IV.  CBIS  EVALUATION  METHODS 

Examining  the  CBIS  evaluation  literature  in  light  of  evaluative  epistemology 
and  methodological  variety  provides  insight  into  the  assumptions  and  strategies 
that  dominant   CBIS   evaluation.   While   it   is   not   the   intent   of  this   article   to 
exhaustively  review  the  CBIS  evaluation  literature,  examining  selected  examples 
of  CBIS   evaluation    illustrates    both    common    and   unusual    CBIS    evaluation 
approaches  and  provides  insight  into  improving  CBIS  evaluation. 
Low  Variety  Methods 

The  most  common  approach  to  CBIS  evaluative  is  to  combine  low  variety 
methods  with  a  positivist  paradigm.  Most  user  surveys  (e.g.,  Franz,  Robey,  and 
Koeblitz,  1986;  Rushinek  and  Rushinek,  1983;  Srinivasan;  1985),  measures  of 
computer  usage  (Ferrari,  1978;  Hiltz  and  Turoff,  1981),  and  other  uses  of  external, 
"objective"  data  sources  (e.g.,  Banker  and  Kauffman,  1988;  Laudon,  1986)  to 
evaluate    computing    activities    combine    low   variety    methods    with    positivist 
assumptions.   Sprague    and    Carlson    (1982)    explicitly    argue    for   a   positivist 
paradigm,  survey  sampling  approach  to  evaluating  decision  support  systems. 

One  striking  feature  of  low  variety,  positivist  evaluation  is  the  relative 
absence  of  measures  of  what  people  do  with  systems  as  opposed  to  what  they 
say  they  do.  While  exceptions  do  exist  (e.g,  Banker  and  Kauffman,  1988;  Laudon, 
1986),  the  majority  of  low  variety  CBIS  evaluation  relies  upon  self-reports  of 
systems  usage  and  usefulness,  rather  than  measures  of  system  usage  and 
usefulness.  The  organizational  and  political  contexts  within  which  systems  are  im- 
plemented provide  strong  motivations  for  organizational  actors  to  appear  highly 
rational,  satisfied,  and  consistent  in  their  relationships  with  organizational  CBIS 
(Feldman  and  March,  1981;  Kling,  1987).  Direct  measures  of  systems  usage  are 
therefore  less  likely  to  be  biased  towards  appearances  of  rationality  than  are  self 
reports  of  system  usage. 
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Combining  low  variety  methods  and  an  interpretive  paradigm  is  unusual, 
since   interpretive   perspectives   generally   assume   complexities   in   computing 
relationships  not  captured  by  low  variety  methods.  However,  Robertson  (in  press) 
illustrates  a  low  variety  method/interpretive  paradigm  approach  to  evaluating 
systems  usage  in  a  management  consulting  firm.  Robertson's  theory  of  systems 
impact  is  strongly  interpretive.  Information  systems  are  socially  determined,  and 
are  dependent  upon  the  shared  meanings  of  subgroups  of  organizational  actors. 
However,    Robertson    uses    structured    self-reports    of   relationships    between 
organizational  actors,  and  sophisticated  statistical  methodology  to  analyze  rela- 
tionships between  organizational  actors.   Results  suggest  partial  support  for 
hypothesized    relationship    between    social    interpretations    of    systems    §nd 
organizational  subgroup. 

Kling  (1987)  cautions  researchers  against  exclusive  use  of  low  variety 
methods  within  an  interpretive  perspective.  Kling  argues  that  low  variety  methods, 
such  as  standardized  surveys,  can  only  be  effectively  combined  with  interpretive 
perspectives  when  high  variety  methods  have  been  previously  used  to  delineate 
important  computing  relationships  in  an  organization.  Kling  argues  that  low  variety 
methods    do    not    provide    sufficient    variety    to    capture    the    subtleties    and 
complexities  that  interpretive  researchers  assume  exist  in  and  around  computing 
relationships  in  organizations. 
High  Variety  Methods 

While  less  frequent,  high  variety  methods  are  employed  to  evaluate  CBIS. 
Examples  of  depth-probe  or  longitudinal  field  research  combined  with  an 
interpretive  paradigm  include  Mann  and  Williams  (1960),  Boland  and  Day  (in 
press),  and  Barley  (1986).  Boland  and  Day  (in  press)  illustrate  interpretive 
evaluation  using  high  variety  methods  from  the  upper  end  of  Figure  2.  They 
conducted  interviews  with  one  credit  union  systems  analyst  over  a  two  year 
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period.  The  first  year's  interviews  were  conducted  while  the  analyst  worked  on  a 
new  loan  application  system,  while  the  second  year's  interviews  consist  of  the 
analyst's  reflections  about  the  previous  year's  systems  design  experience.  These 
interviews  reveal  that  systems  design  may  require  more  moral,  ethical  choices  by 
systems  analysts  than  is  generally  recognized.  For  example,  the  systems  analyst 
was  asked  to  design  an  easy-to-use  system  to  allow  poorly  paid  workers  to  re- 
place higher  pay,  higher  talent  workers. 

Articles    suggesting    CBIS    evaluation    strategies    for    IS    practitioners 
frequently  reflect  a  high  variety  /  positivist  perspective  to  evaluation  (e.g.,  Burch 
and  Grudnitski  1986).  For  example,  Keen  (1975,  1981)  argues  that  consideration 
of  qualitative  benefits  should  be  the  most  important  consideration  in  evaluating 
systems  and  that  high  variety  methods  are  appropriate  for  evaluating  such 
benefits.  However,  Keen  argues  for  high  variety  methods  within  a  framework  of 
positivist  assumptions  that  include  detachment  between  evaluator  and  evaluation, 
a  priori  specification  of  "success"  and  of  system  goals,  and  objective  analysis  of 
data.  Existing  evidence  suggests  that,  while  high  variety/positivist  perspectives 
for  CBIS  evaluation  are  frequently  recommended  in  the  IS  literature,  they  are 
seldom  applied  by  their  assumed  constituency  of  IS  practitioners  (Hogue  and 
Watson,  1984;  Sprague  and  Carlson,  1982).  One  explanation  for  the  apparent 
lack  of  interest  in  high  variety/positivist  approaches  to  evaluation  may  be  that  IS 
professionals  do   not  share    the   epistemological   convictions   of  the   positivist 
research  paradigm.  For  example,  IS  professionals  may  be  unconcerned  with 
obtaining   controlled,    objective    observations,    that   are    mediated   by   a   priori 
classification. 
"Mixed"  Methods 

Geertz  (1983)  has  observed  that  once-distinct  boundaries  between  social 
science  disciplines  are  increasingly  blurred,  and  that  mixing  of  methods  and 
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perspectives  is  emerging  as  a  dominant  research  trend.  This  mixing  of  methods 
and  perspectives  is  reflected  in  the  increasing  use  of  mixed  methods  and  mixed 
paradigms  in  CBIS  evaluation.  "Mixed"  CBIS  evaluations  have  combined  a  variety 
of  methodologies  including  questionnaires,  financial  and  budgetary  data  analysis, 
unstructured     interviews,     examination     of     documents     and     memoranda, 
observations  at  meetings,  and  tape  recordings  to  understand  systems  impact. 
The  mixing  of  high  and  low  variety  methodologies  is  frequently  combined  with  a 
mixing  of  paradigms,  leading  to  explorations  of  the  implications  of  contrasting 
paradigmatic  assumptions.  Two  recent  examples  illustrate  mixing  paradigmatic 
assumptions  and  evaluation  methods. 

Franz  and  Robey  (1984)  studied  the  impact  of  an  automobile  insurance 
company  information  system  over  a  22  month  period,  using  a  combination  of 
open-ended  interviews,  questionnaires,  examination  of  critical  incident  files,  formal 
documents,  and  observations  at  meetings.  Their  paradigmatic  perspective  is 
largely  interpretive,  but  also  involves  exploring  the  extent  to  which  positivist 
theories  can  explain  observed  systems  development  behavior.  Their  theoretical 
perspective    compares    and   contrasts    rational    and   political    explanations   for 
observed  systems  design  behavior.  Both  quantitative  and  qualitative  data  is 
gathered  to  motivate  and  interpret  theoretical  propositions. 

Kaplan  and  Duchon  (1988)  combined  low  and  high  variety  methodologies 
to  evaluate  the  impact  of  a  new  computerized  data  management  in  a  medical 
laboratory.    Methods    employed    include    open-ended    interviews,    sustained 
observation,  and  surveys.  Both  quantitative  and  qualitative  data  analyses  were 
used  to  evaluate  systems  impact  and  success,  and  both  interpretive  and  positivist 
assumptions    are    explicated,    discussed,    and    evaluated.    Based    upon    their 
combined  qualitative  and  quantitative  data  analyses,  Kaplan  and  Duchon  propose 
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a  largely  interpretive  model  to  explain  changes  in  job  characteristics  and  work 
relationships  resulting  from  computerization. 
V.  IMPLICATIONS  FOR  CBIS  EVALUATION 

Viewing     CBIS     evaluation     within     the     framework     of     paradigmatic 
assumptions   and   evaluation   methodology   suggests   several    implications   for 
improving   CBIS    evaluation.   In   particular,    four   propositions    emerge.   These 
observations  pertain  to  recognizing  implicit  values,  beliefs,  and  assumptions, 
acknowledging  trade  offs,  building  complicated  descriptions  of  CBIS,  and  using 
positivist  and  interpretive  evaluation  as  interdependent,  not  competing,  modes  of 
inquiry. 

Proposition  #1  -  CBIS  evaluations  always  contain  implicit  assumptions, 
values,  and  beliefs. 

Any   evaluation   of  a   CBIS    contains    the    implicit   values,    beliefs,    and 
assumptions   of  the   evaluator.   Positivist   evaluators   believe    in   separation   of 
researcher  and  subject,  the  importance  of  objective,  generalizable  knowledge, 
and  quantitative  methodologies.  Interpretive  evaluators  belief  in  immersing  the 
researcher  in  the  subject,  the  importance  of  particular,  subjective  knowledge,  and 
qualitative  methodologies.  While  positivist  assumptions  can  be  mixed  with  inter- 
pretive assumptions,  assumptions,  values,  and  beliefs  cannot  be  eliminated  from 
the  process  of  CBIS  evaluation. 

However  hidden,  implicit  assumptions  cannot  be  argued,  debated,  or 
scrutinized.  Consequently,  effective  presentation  of  CBIS  evaluation  research 
requires  explicitly  recognizing  and  acknowledging  one's  assumptions,  values, 
and  beliefs.  Ultimately,  the  issue  is  not  whether  or  not  researchers  bring  values, 
assumptions  and  beliefs  to  CBIS  evaluation,  but  whether  these  values, 
assumptions,  and  beliefs  are  defenselessly  recognized,  acknowledged,  and  dis- 
cussed. 
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Proposition  #2  -  CBIS  evaluators  always  make  trade  offs. 

The  choice  of  an  evaluation  method  or  perspective  always  involves  trade 
offs.  To  see  this,  consider  Thorngate's  postulate  of  commensurate  complexity 
(Thorngate,  1976;  Weick,  1979,  1984) .  Thorngate  observes  that  it  is  impossible  to 
generate  explanations  of  behavior  that  are  simultaneously  general,  accurate,  and 
simple.  At  best,  two  of  the  three  objectives  can  be  met.  Movement  toward  any  two 
objectives  is  movement  away  from  the   third.  Therefore,  as  a  theory  moves 
towards  greater  generality  and  simplicity,  it  necessarily  becomes  less  able  to 
predict  specific  behavior  (i.e.,  less  accurate).  As  a  theory  moves  towards  greater 
accuracy  and  simplicity,  it  becomes  less  able  to  describe  behavior  in  general.  And 
as  a  theory  becomes  more  general  and  accurate,  it  loses  simplicity. 

The  implication  of  Thorngate's  postulate  for  CBIS  evaluators  is  that  the 
choice  of  a  method  and  perspective  in  evaluating  a  CBIS  is  a  forced  trade  off 
between  the  competing  goals  of  accuracy,  generality,  and  simplicity.  While  CBIS 
evaluators  may  differ  as  to  which  goals  are  of  greater  importance,  trade  offs  are 
inevitable.  Consciously  acknowledging  these  trade  offs,  and  openly  discussing 
the  value  of  alternative  combinations  of  goals  is  likely  to  lead  to  an  improved 
understanding  of  the  merits  of  alternative  perspectives  and  approaches  to  CBIS 
evaluation. 


Proposition  #3  -  'Complicated'  evaluations  of  CBIS  are  likely  to  be  more 
accurate. 


Boulding  (1956)  and  Pondy  and  Mitroff  (1978)  have  observed  that  human 
organizations   are   among   the   most   complex   of  known   systems.   Effectively 
evaluating  CBIS   requires  understanding  the  relationship  between  computing 
technology  and  complex,  on-going  human  organizations.  Recent  approaches  to 
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CBIS  evaluation  attempt  to  understand  and  describe  the  complexity  of  human 
organizations  by  using  multiple  evaluation  methods,  and  by  mixing  assumptions 
from  positivist   approaches  with   those  of  interpretive   approaches.   Such   ap- 
proaches are  designed  to  reflect  the  complexity  of  human  organizations  by 
providing  "complicated  descriptions"  (Weick,  1979)  of  the  impact  of  computing 
technology  on  organizations. 

Triangulating   evaluation   methods   provides   one    means   of  generating 
complicated  descriptions  of  systems.  Triangulation  involves  combining  high  and 
low  variety  methodologies  to  study  a  phenomena  (Denzin,  1978).  Researchers 
using  multiple  methods  can  be  more  confident  in  their  findings,  are  more  likely  to 
discover  unusual  or  unexpected  phenomena,  and  can  more  effectively  synthesize 
and  integrate  disparate  theories  or  paradigms  (Jick,  1979).  However,  the  use  of 
multiple  measures  is  not  a  sufficient  condition  for  achieving  triangulation  in 
measurement  (Webb,  Campbell,  Schwartz,  and  Sechrest,  1966).  For  example,  the 
use  of  multiple  survey  questions  regarding  user  satisfaction  does  little  to  achieve 
measurement  triangulate,  since  such  measures  are  largely  redundant  (Rushinek 
and  Rushinek,  1983).  Rather,  triangulating  measurement  requires  choosing  low 
and  high  variety  methods  from  Figure  1. 

An  alternative  approach  to  generating  "complicated  descriptions"  is  to 

triangulate  theory.  Achieving  theoretical  triangulation  requires  holding  conflicting 

assumptions   regarding   the   impact   of  a   CBIS.   Simultaneously   trusting   and 

doubting  the  same  assumption  achieves  a  theoretical  triangulation  that  can  be 

matched  with  methodological  triangulation.  As  Weick  observes: 

Any  person  who  has  a  view  of  the  world  and  who  also  discredits 
part  of  that  view  winds  up  with  two  ways  to  examine  a  situation. 
Discrediting  is  a  way  to  enhance  requisite  variety  and  a  way  to  reg- 
ister more  of  the  varietv  that's  present  in  the  world  (Weick,  1979,  p. 
228). 
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Two  methods  exist  for  generating  and  holding  conflicting  assumptions  and 
theoretical  perspectives.  One  is  to  triangulate  investigators  by  creating  a  multi- 
perspective  team  with  differing  ontological  and  epistemological  assumptions 
(Denzin,  1978;  Legge,  1984).  Such  an  approach  to  triangulation  is  used  by  Kaplan 
and    Duchon    in    their    investigation    of   computing    in    a    medical    laboratory. 
Alternatively,  CBIS  evaluators  can  make  themselves  more  "complicated,"  by  con- 
sciously generating  and  explicating  alternative,  conflicting  theoretical  perspectives 
(Bartunek,  Gordon,  and  Weathersby,  1983;  Streufert  and  Swezey,  1986).  Using 
either  approach  to  achieving  theoretical  triangulation  increases  the  likelihood  of 
producing  accurate  descriptions  of  the  effects  of  CBIS  evaluation  and  reduces  the 
chance  that  evaluators  will  "see"  only  what  they  expect  to  see. 

Proposition  #4  -  Interpretive  and  positivist  evaluation  are  interdependent, 
not  competing,  modes  of  inquiry. 

Positivist  and  interpretive  approaches  to  science  are  often  viewed  as 
competing,  unreconcilable  paradigms  for  conducting  research  (e.g.,  Chua  1986; 
Legge,  1984).  However,  achieving  theoretical  triangulation  in  evaluation  requires 
viewing    interpretive    and    positivist    approaches    not    as    competing,    but    as 
complementary  modes  of  inquiry  (Johnson,  1989).  In  conducting  "complicated" 
evaluation,   evaluators  focus  exclusive  attention  on  interpretive   and  positivist 
paradigms  only  temporarily,  and  move  between  paradigms  in  order  to  generate 
more  accurate  descriptions. 

Why  might  cycling  between  interpretive  and  positivist  evaluation  lead  to 
more  accurate  descriptions  of  the  effects  of  CBISs?  One  explanation  for  the 
effectiveness  of  interdependent,  concurrent  modes  of  inquiry  may  be  that  such 
approaches  reflect  a  "natural"  division  of  responsibility  in  the  brain.  Research 
suggests  that  the  left  and  right  brain  hemispheres  of  humans  utilize  distinct 
approaches  to  acquiring  information  about  the  world  (Meyer,  1985;  Simon,  1977; 
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Taggert  and  Robey   1981).  Levy  summarizes  the  considerable   research  with 
subjects  whose  brain  hemispheres  have  been  separated  as  follows: 

Each  side  of  the  brain  is  able  to  perform  and  chooses  to  perform  a 
certain  set  of  cognitive  tasks  which  the  other  side  finds  difficult  or 
distasteful  or  both...  The  right  hemisphere  synthesizes  over  space. 
The  left  hemisphere  analyzes  over  time.  The  right  hemisphere  notes 
visual  similarities  to  the  exclusion  of  conceptual  similarities.  The  left 
hemisphere  does  the  opposite.  The  right  hemisphere  perceives 
form,   the   left   hemisphere,   detail.   The   right   hemisphere   codes 
sensory  input  in  terms  of  images,  the  left  hemisphere  in  terms  of 
linguistic  descriptions.  The  right  hemisphere  lacks  a  phonological 
analyzer;  the  left  hemisphere  lacks  a  Gestalt  svnthesizer  (Levy, 
1974,  p.  167). 

Moving  between  interpretive  and  positivist  assumptions  in  CBIS  evaluation 
may  generate  more  accurate  descriptions  of  organizational  phenomena  because 
brain  physiology  loosely  corresponds  to  the  alternative  modes  of  inquiry  found  in 
interpretive  and  positivist  approaches  to  evaluation.  The  right  brain  hemisphere  is 
concerned  with   experiential,   impressionistic   understanding,   while   the   left   is 
concerned  with  logic  and  analysis.  Similar  observations  can  be  made  with  respect 
to  the  processes  of  knowledge  acquisition  used  in  interpretive  and  positivist 
modes  of  inquiry.  Interpretive  inquiry  largely  relies  on  experiential  approaches  to 
acquiring    knowledge    while     positivist    inquiry    relies    on    analytical,     logical 
approaches. 
VI.  CONCLUSION 

Banville  and  Landry  (1989),  building  on  the  work  of  Whitley  (1984)  have 
recently    described    the    MIS    research    field    as    a    "fragmented    adhocracy," 
characterized  by:   (1)  distinct,  independent  schools  of  thought,  (2)  little  stan- 
dardization as  to  methods  and  measures,  (3)  problems  of  broad  scope  that  defy 
standardization.  Culnan's  co-citation  analysis  of  MIS  research  supports  the  view 
of  MIS  as  a  multiparadigmatic  discipline.  She  identifies  nine  subfields  of  MIS  in 
one  study  (Culnan,  1986)  and  five  in  a  subsequent  one  (Culnan  and  Swanson, 
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1987).  Banville  and  Landry  suggest  that  fruitful  inquiry  in  a  fragmented  adhocracy 
is  fueled  by  applying  multiple  perspectives  and  methods  to  complex,  ill-defined 
problems. 

The    CBIS    evaluation    literature    reflects    the    "fragmented    adhocracy" 
characteristics  of  the  more  general  MIS  literature.  The  CBIS  evaluation  literature 
shows  distinct  schools  of  thought  (e.g.,  positivist  and  interpretive),  little  agreement 
as  to  appropriate  methods  and  measures  (e.g.,  high  and  low  variety),  and  a 
concern  with  tackling  broad,  encompassing  problems  that  defy  standardized 
formulation  (e.g.,  the  value  of  CBIS).  The  existence  of  a  "fragmented  adhocracy" 
in  CBIS  evaluation  suggests  that  disparate  approaches  will  be  used  to  evaluate 
CBIS.    However    generating    productive,    consequential    scholarship    requires 
sufficient  cohesion  among  investigators  that  research  findings  can  be  shared  and 
understood  (Weick,  1983). 

One  key  element  of  cohesion  that  can  be  built  into  CBIS  evaluation  is 
shared    understanding    of   the    values,    assumptions,    and    beliefs    implicit    in 
alternative  approaches  to  CBIS  evaluation.  The  plurality  of  perspectives  within 
MIS  insures  that  CBIS  evaluation  will  be  examined  from  a  variety  of  perspectives. 
However,  effectively  discerning  the  relative  contributions  of  individual  research 
efforts    requires    understanding    alternative    epistemological    assumptions    and 
limitations.    By    defenselessly    explicating    and    discussing    these    underlying 
assumptions,  CBIS  evaluators  can  understand  one  another  more  clearly,  thereby 
increasing  the  likelihood  of  improving  our  understanding  of  the  impact  of  systems 
on  organizations,  and  of  organizations  on  systems. 


The  Epistemology  of  Evaluation 

24 


REFERENCES 

Banker,    R.D.    and    R.J.    Kauffman,    "Strategic    Contributions    of    Information 
Technology:  An  Empirical  Study  of  ATM  Networks,"  Proceedings  of  the 
Ninth  International  Conference  on  Information  Systems.  Minneapolis,  1988. 

Banville,    C.    and    M.    Landry,    "Can    the    Field    of    MIS    be    Disciplined?," 
Communications  of  the  ACM.  32,  1  (1989),  48-60. 

Barley,  S.R.,  "Technology  as  an  Occasion  for  Structuring:  Evidence  from 
Observations  of  CT  Scanners  and  the  Social  Order  of  Radiology 
Departments,"  Administrative  Science  Quarterly.  31  (1986),  78-108. 

Bartunek,  J.M.,  J.R.  Gordon,  and  R.P.  Weathersby,  "Developing  'Complicated' 
Understanding  in  Administrators,"  Academy  of  Management  Review.  8,  2 
(1983),  273-284. 

Bernstein,  R.J.,  Praxis  and  Action.  Philadelphia:  University  of  Pennsvlvania  Press, 
1971. 

Bernstein,  R.J.,  The  Restructuring  of  Social  and  Political  Theory.  Oxford:  Basil 
Blackwell,  1983. 

Blackler,  F.H.  and  Brown,  C.A.,  "Qualitative  Research  and  Paradigms  of  Practice," 
Journal  of  Management  Studies.  20,  3  (1983),  349-365. 

Boland,    R.J.,    Jr.,    and    W.    Day,    "The    Experience    of    System    Design:    A 
Hermeneuitics    of    Organizational    Action,"     Scandanavian    Journal     of 
Management,  in  press. 

Boulding,  K.E.,  "General  Svstems  Theory:  The  Skeleton  of  Science,"  Management 
Science,  2(1956),  197-207. 

Bowen,  W.,  "The  Puny  Payoff  From  Office  Computers,"  Fortune  (Mav  26,  1986), 
20-24. 

Bozcany,  W.J.,  "Justifying  Office  Automation,"  Journal  of  Systems  Management. 
34,7(1983),  15-19. 

Burch,  J.G.  and  G.  Grudnitski,  Information  Systems:  Theory  and  Practice.  New 
York:  Wiley  and  Sons,  1986. 

Burrell,  G.  and  G.  Morgan,  Sociological  Paradigms  and  Organisational  Analysis. 
London:  Heinemann,  1979. 

Business  Week.  "The  Productivity  Paradox  -  A  Special  Report,"  (June  6,  1988), 
100-114. 

Caddell,  D.D.,  "The  Limited  Computer  Feasibility  Study:  A  Cost-saving  Approach," 
Journal  of  Accountancy.  (April,   1985),  122-126. 


The  Epistemology  of  Evaluation 

25 

Campbell,   D.T.,   '"Degrees  of  Freedom'   and   the   Case   Study,"  Comparative 
Political  Studies.  (July,  1975),  178-193. 

Chua,  W.F.,  "Radical  Developments  in  Accounting  Thought,"  The  Accounting 
Review.  61,  4  (October,  1986),  601-632. 

Conant,  R.C.  and  R.W.  Ashby,  "Every  Good  Regulator  of  a  System  Must  be  a 
Model  of  That  System,"  International  Journal  of  Systems  Science.  1,  2 
(1970),  89-97. 

Culnan,  M.J.,  "The  Intellectual  Development  of  Management  Information  Systems. 

156-172. 

Culnan,  M.J.  and  E.B.  Swanson,  "Mapping  the  Intellectual  Structure  of  MIS.  1980- 
1985:  a  Co-citation  Analysis,"  MIS  Quarterly.  11,  3  (1987),  340-353. 

Daft,  R.L.,  and  J.C.  Wiginton,  "Language  and  Organization,"  Academy  of 
Management  Review.  6,  3  (1979),  385-395. 

DeMey,  M.,  The  Cognitive  Paradigm.  Boston:  D.  Reidel,  1982. 

Denzin,  N.K.,  The  Research  Act.  2nd  edition,  New  York:  McGraw-Hill,  1978. 

Douglas,  J.D.,  Investigative  Social  Research.  Beverly  Hills:  Sage  Publications, 
1976. 

Evered,   R.    and   M.R.   Louis,   "Alternative   Perspectives   in   the    Organizational 
Science:    'Inquiry   From   the    Inside'    and   'Inquiry   From   the   Outside,'" 
Academy  of  Management  Review.  6,  3  (1981),  385-395. 

Feldman,  M.S.  and  J.G.  March,  "Information  in  Organizations  as  Signal  and 
Symbol,"  Administrative  Science  Quarterly.  26,  2  (June  1981),  171-186. 

Ferrari,  D.,  Computer  Systems  Performance  Evaluation.  Englewood  Cliffs,  NJ: 
Prentice-Hall,  1978. 

Feyerabend,   P.,    Against    Method:    An   Outline    of   an   Anarchistic   Theory   of 
Knowledge.  London:  New  Left  Books,  1975. 

Franz  ,  C.  and  D.  Robey,  "An  Investigation  of  User-led  ystems  Design:  Rational 
and  Political  Perspectives,"  Communications  of  the  ACM.  27,  12  (1984), 
1202-1209. 

Franz  ,  C.  and  D.  Robey,  "Strategies  for  Research  on  Information  Systems  in 
Organizations.  A  Critical  Analysis  of  Research  Purpose  and  Time  Frame," 
in  R.J.  Boland  and  R.A.  Hirschheim  (Eds.)  Critical  Issues  in  Information 
Systems  Research.  New  York:  Wiley,  1987. 

Franz,    C.R.,    D.    Robey,    and    R.R.    Koeblitz,    "User   Response    to    an   Online 
Information  System:  A  Field  Experiment,"  MIS  Quarterly.  10  (1986),  29-42. 


The  Epistemology  of  Evaluation 

26 

Geertz,  C,  "Blurred  Genres:  the  Refiguration  of  Social  Thought,"  Chapter  1  in 
Geertz,  C.,  Local  Knowledge:  Further  Essays  in  Interpretive  Anthropology. 
New  York:  Basic  Books,  1983,  19-35. 

Gremillion,    L.L.    and    P.J.    Pyburn,    "Justifying    Decision    Support    and    Office 
Automation  Systems,"  Journal  of  Management  Information  Systems.  2,  1 
(1985),  5-17. 

Habermas,  J.,  Knowledge  and  Human  Interest.  2nd  Edition,  London:  Heinemann, 
1978. 

HaiiLilloii,  S.  and  N.L.  Chervany,  "Evaluating  Information  System  Effectiveness  - 
Part  I:  Comparing  Evaluation  Approaches,"  MIS  Quarterly.  (September, 
1981a),  55-69. 

Hamilton,  S.  and  N.L.  Chervany,  "Evaluating  Information  System  Effectiveness  - 
Part  II:    Comparing  Evaluator   Viewpoints,"    MIS   Quarterly.    (December, 
1981b),  55-69. 

Heidegger,  M.,  Being  and  Time.  New  York:  Harper,  1962. 

Hiltz,  S.R.  and  M.  Turoff,  "The  Evolution  of  User  Behavior  in  a  Computerized 
Conferencing  System,"  Communications  of  the  ACM.  24,  11  (1981),  739- 
751. 

Hirschheim,  R.A.  and  S.  Smithson,  "A  Critical  Analysis  of  Information  Systems 
Evaluation,"  in   N.   Bjorn-Andersen  and   G.B.   Davis   (Eds.),   Information 
Systems  Assessment:  Issues  and  Challenges.  Amsterdam:  North-Holland, 
1988. 

Hogue,  J.,  and  Watson,  H.,  "Current  Practices  n  the  Development  of  Decision 
Support  Systems,"  Proceedings  of  the  Fifth  International  Conference  on 
Information  Systems.  Tucson,  1984,  117-127. 

Jick,  T.D.,  "Mixing  Qualitative  and  Quantitative  Methods:  Triangulation  in  Action," 
Administrative  Science  Quarterly.  24  (1979),  602-611. 

Johnson,  O.,  "Accounting  Research  Education:  View  or  Vision,"  unpublished 
working  paper,  University  of  Illinois,  1989. 

Kaplan,  B.  and  D.  Duchon,  "Combining  Qualitative  and  Quantitative  Methods  in 
Information  Systems   Research:   A  Case  Studv,"   MIS   Quarterly.   12,  4 
(1988),  571-587. 

Keen,  P.W.,  "Computer-Based  Decision  Aids:  The  Evaluation  Problem,"  Sloan 
Management  Review.  16,  3  (Spring,  1975),  17-30. 

Keen,  P.W.,  "Value  Analysis:  Justifying  Decision  Support  Systems,"  MIS  Quarterly 
(March,  1981),  1-14. 

Kling,  R.,  "Defining  the  Boundaries  of  Computing  Across  Complex  Organizations," 
in   R.J.    Boland,   Jr.    and    R.    A.    Hirschheim    (Eds.),    Critical    Issues    in 
Information  Systems  Research.  New  York:  Wiley,  1987. 


The  Epistemology  of  Evaluation 

27 

Kling,  R.  and  S.  Iacono,  "The  Control  of  Information  Systems  Developments  After 
Implementation,"  Communications  of  the  ACM.  27,  12  (1984),  1218-1226. 

Kuhn,  T.S.,  The  Structure  of  Scientific  Revolution  (2nd  edition),  Chicago: 
University  of  Chicago  Press,  1970. 

Lakatos,     I.     "Falsification     and     the     Methodology     of    Scientific     Research 
Programmes,"  in  I.  Lakatos  and  A.  Musgrave  (Eds.),  Criticism  and  the 
Growth  of  Knowledge.  Cambridge:  Cambridge  University  Press,  1970. 

Laudon,  K.,  "Data  Quality  and  Due  Process  in  Large  Interorganizational  Record 
Systems,"  Communications  of  the  ACM.  29,  1  (1986),  4-11. 

Legge,  K.,  Evaluating  Planned  Organizational  Change.  New  York:  Academic 
Press,  1984. 

Levy,  J.,  "Psychobiological  Implications  of  Bilateral  Asymmetry,"  in  S.J.  Dimond 
and  J.G.  Beaumont  (Eds.),  Hemisphere  Function  in  the  Human  Brain.  New 
York:  Wiley,  1974,  121-183. 

Mann,  F.C.  and  L.K.  Williams,  "Observations  on  the  Dynamics  of  a  Change  to 
Electronic  Data-Processing  Equipment,"  Administrative  Science  Quarterly. 
5  (1960),  217-256. 

Mann,  F.C.  and  L.R.  Hoffman,  Automation  and  the  Worker:  A  Study  of  Social 
Change  in  Power  Plants.  New  York:  Wiley,  1960. 

Markus,    M.L.    and    D.    Robey,    "Information   Technology    and    Organizational 
Change:  Causal  Structure  in  Theory  and  Research,"  Management  Science. 
34,  5  (May,  1988),  583-598. 

Meals,  D.W.,   "Svstems   Evaluation,"  Journal  of  Systems   Management.   (July, 
1977),  6-9. 

Meyer,  A.  D.,  "Iconic  Data  in  Organizational  Research,"  unpublished  working 
paper,  University  of  Oregon,  1985. 

Morgan,  G.  and  L.  Smircich,  "The  Case  for  Qualitative  Research,"  Academy  of 
Management  Journal.  5,  4  (1980),  491-500. 

Patton,  M.Q.,  How  to  Use  Qualitative  Methods  in  Evaluation.  Beverly  Hills:  Sage 
Publications,  1987. 

Pfeffer,  J.,  Organizational  Design.  Arlington  Heights,  IL:  AHM  Publishing,  1978. 

Pondy,  L.R.  and  I.I.  Mitroff,  "Beyond  Open  Systems  Models  of  Organization,"  in 
B.M.  Staw  (Ed.)  Research  in  Organizational  Behavior.  Greenwich,  Conn.: 
JAI  Press,  1978. 

Pondy,  L.R.,  "The  Two  Faces  of  Evaluation,"  in  H.W.  Melton  and  D.J.H.  Watson 
(Eds.),  Interdisciplinary  Dimensions  of  Accounting  for  Social  Goals  and 
Social  Organizations.  Columbus,  OH:  Grid  Inc.,  1977. 


The  Epistemology  of  Evaluation 

28 

Popper,  K.R.,  Conjectures  and  Refutations.  3rd  Edition  London:  Routledge  and 
Kegan  Paul,  1972a. 

Popper,  K.R.,  The  Logic  of  Scientific  Discovery.  London:  Hutchinson,  1972b. 

Robertson,  D.C.,  "Social  Determinants  of  Information  Systems  Use,"  Journal  of 
MIS,  in  press. 

Russell,  B.,  A  History  of  Western  Philosophy.  New  York:  Simon,  1945. 

Rushinek,  A.,  and  S.F.  Rushinek,  "An  Evaluation  of  Mini/Micro  Systems:  An 
Empirical  Multivariant  Analysis,"  Database  (Summer,  1983),  37-47. 

Schutz,  A.,  Collected  Papers.  Vol.  1:  The  Problem  of  Social  Reality.  (M.  Natanson 
Ed.),  The  Hague:  Nijhoff,  1962. 

Scott,  W.R.,  S.M.  Dornbusch,  B.C.  Busching,  and  J.D.  Laing,  "Organizational 
Evaluation  and  Authority,"  Administrative  Science  Quarterly.  12  (1967),  93- 
117. 

Simon,  H.A.,  "On  the  Forms  of  Mental  Representation,"  working  paper  #344, 
Carnegie-Mellon  University,  1977. 

Sprague,  R.H.,  Jr.  and  E.D.  Carlson,  Building  Effective  Decision  Support  Systems. 
Englewood  Cliffs,  NJ:  Prentice-Hall,  1982. 

Srinivasan,  A.,  "Alternative  Measures  of  System  Effectiveness:  Associations  and 
Implications,"  MIS  Quarterly.  9,  3  (September,  1985),  243-253. 

Strassman,  P.,  Information  Payoff:  The  Transformation  of  Work  in  the  Electronic 
Age.  New  York:  Free  Press,  1985. 

Streufert,  S.  and  R.W.  Swezey,  Complexity.  Managers,  and  Organizations.  New 
York:  Academic  Press,  1986. 

Taggart,  W.  and  D.  Robey,  "Minds  and  Managers:  On  the  Dual  Nature  of  Human 
Information   Processing   and    Management,"   Academy   of   Management 
Review,  6(1981),  187-195. 

Thorngate,  W.,  "'In  General'  Vs.  'It  Depends,':  Some  Comments  on  the  Gergen- 
Schlenker  Debate,"  Personality  and  Social  Psychology  Bulletin.  2  (1976), 
404-410. 

Toulmin,  S.,  Human  Understanding.  Vol.  1.  Princeton,  NJ:  Princeton  Universitv 
Press,  1972. 

Uttal,  Bro,  "What's  Detaining  the  Office  of  the  Future?,"  Fortune  (Mav  3,  1982), 
176-196. 

Webb,  E.J.,  D.T.  Campbell,  R.D.  Schwartz,  and  L.  Sechrest,  Unobtrusive 
Measures.  Chicago,  IL.:  Rand  McNally,  1966. 

Weick,  K.E.,  The  Social  Psychology  of  Organizing.  Reading,  MA:  Addison- Wesley, 
1979. 


The  Epistemology  of  Evaluation 

29 

Weick,  K.E.,  "Contradictions  in  a  Community  of  Scholars:  the  Cohension- 
Accuracy  Tradeoff,"  The  Review  of  Higher  Education.  6,  4  (1983),  253-267. 

Weick,  K.E.,  "Theoretical  Assumptions  and  Research  Methodology  Selection,"  in 
F.W.  McFarlan  (Ed.),  The  Information  Systems  Research  Challenge. 
Boston,  MA:  Harvard  Business  School  Press,  1984. 

Whitley,  R.,  The  Intellectual  and  Social  Organization  of  the  Sciences.  Oxford: 
Clarendon  Press,  1984. 

Zmud,  R.W.  and  L.E.  Apple,  "Measuring  the  Infusion  of  a  Multi-Business  Unit 
Innovation,"  unpublished  working  paper,  Florida  State  University,  1989. 


HECKMAN       |±| 
BINDERY  INC.        |§§| 

JUN95 

Bound  -To -Plea»^   N  MANCHESTER. 
INDIANA  46963 

